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ABSTRACT 


Modern  weapons  systems  rely  upon  complex  electronics  for 
mission  essential  functions,  such  as  fire  control.  The 
repair  strategy  is  to  remove  and  replace  a  faulty  ‘black 
box"  (known  as  a  Line  Replaceable  Unit  or  LRU)  at  the 

forward  weapon  site,  and  evacuate  it  for  repair.  However, 
on-board  diagnostics  and  methods  available  at  the  owning 
unit  have  proved  inadequate  in  isolating  a  fault  to  a  single 
LRU.  This  leads  to  a  strategy  of  'swapping'  known  good 

components  for  those  in  an  ambiguity  group  to  return 
the  system  to  operational  status. 

The  Army's  program  to  solve  this  identified  problem  is  the 
Intermediate  Forward  Test  Equipment  (IFTE) ,  comprised  of  a 
van  mounted  Base  Shop  Test  Facility  (BSTF)  for  intermediate 
level  diagnosis  and  repair  of  evacuated  modules,  and  the 
Contact  Test  Set  (CTS)  and  portable  tester  designed  to  be 
employed  by  the  intermediate  maintenance  support  team  at  the 
weapon  system  site  when  the  owning  unit  has  failed  to  fault 
isolate  to  the  LRU  level  with  available  methods.  As  such, 
the  CTS  is  primarily  a  replacement  for  existing  bulky  and 
unwieldy  automatic  test  equipment  (ATE)  and  inadequate 
troubleshooting  procedures  contained  in  technical  manuals. 

The  BSTF  is  used  in  a  bench  test  environment,  and  it  is 
possible  to  apply  human  factors  standards  (for  example, 
those  in  MI L- STD- 1 472 )  to  these  workspaces  (such  as  a  shop 
repair  van) .  However,  such  is  not  the  case  for  portable 
computers  (such  as  the  Contact  Test  Set)  to  be  used  at  the 
weapon  system  site:  for  some  supported  weapons  systems, 

this  will  be  outdoors,  with  the  CTS  positioned  on  or  about 
the  weapon  system.  In  addition  to  such  human  factors 
considerations  as  visibility  of  the  material  on  the  visual 
display  screen  in  various  lighting  conditions,  and  position 
and  posture  of  the  computer  user,  the  diagnostic  aid  used  at 
the  weapon  system  site  becomes  closely  entwined  with  the 
diagnostician’s  task.  Hence,  design  of  a  portable  aid  must 
take  into  account  the  diagnostic  process.  in  order  to 
allocate  functions  to  the  aid  in  an  optimum  manner. 
Considerations  of  hardware  and  software  design  to  implement 
those  functions  follow. 

The  next  generation  of  the  CTS  (often  referred  to  as  the 
Portable  Maintenance  Aid  (PMA))  is  planned  to  house 
diagnostic  expert  systems,  some  of  which  have  been  tested. 
The  promise  of  expert  systems  is  to  make  widely  available 
the  expertise  of  the  best  diagnosticians,  and  to  reduce 
diagnosis  time. 

In  developing  a  Portable  Maintenance  Aid  as  the  next 
generation  of  the  Contact  Test  Set,  the  objective  is  to 
improve  human  performance  of  the  diagnostic  task  at  the 
forward  weapon  system  site.  Due  to  the  intrusive  nature  of 
the  aid  to  the  diagnostic  task,  the  aid  must  be  properly 
designed  from  a  human  factors  perspective  to  achieve  the 
potential  of  both  the  diagnostician  and  the  diagnostic  aid. 
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A  problem  has  been  perceived  in  diagnostic  support  for 
complex  modern  weapons  systems,  particularly  for  electronic 
components.  Each  of  the  services  has  undertaken  development 
of  diagnostic  aids  to  address  this  need  (Demers  (1989,  May, 
Sep),  Jaszka  (1989)).  To  evaluate  the  design  of  such  a 
device  (specifically,  the  Army's  requirement  for  portable 
testing  and  diagnosis  at  the  forward  weapons  system  site)  , 
it  is  necessary  to  understand  the  process  of  diagnosis  and 
current  research  in  the  field  of  diagnostic  job  performance 
aids. 


The  way  a  diagnostic  aid  is  developed  is  based  upon  certain 
assumptions  about  hew  diagnosis  is  or  should  be  performed, 
and  what  types  of  knowledge  the  diagnostician  needs  at  the 
job  site.  Job  knowledge  acquisition  is  divided  between 
training  and  what  is  provided  by  the  job  performance  aid. 
Hence,  philosophies  of  training  are  inherent  in  any 
diagnostic  aid  design.  Because  of  the  interrelationships 
between  training,  personnel  selection  and  advancement,  job 
aid  design,  job  knowledge  and  content,  and  job  satisfaction, 
these  assumptions  inherent  in  the  design  of  a  job  aid  must 
be  made  explicit. 

Unfortunately,  there  has  been  inadequate  research  on 
diagnosis  itself,  including  how  the  human  successfully 
diagnoses  faults,  how  to  best  train  for  diagnosis,  and  how 
to  select  individuals  who  are  likely  to  be  proficient 
diagnosticians . 

REVIEW  OF  RESEARCH 

Knowledge: 

Neiderman  (1988)  and  Morris  (1985)  provide  surveys  of 
research  on  troubleshooting  and  troubleshooting  aids.  Field 
studies  have  also  been  conducted  (Harz  (1981) ,  Nauta  (1985) , 
Fenwick  (1982)).  One  of  the  central  questions  to  the 
development  of  diagnostic  job  performance  aids  and  training 
philosophies  is  the  degree  of  cognitive  or  theoretical 
knowledge  required  by  the  diagnostician  and  the  amount  of 
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procedur a 1 i za t i on  of  the  job  aid.  Research  results  is  this 
area  might  be  viewed  as  contrad i ctory  or  inconclusive. 

For  example,  there  have  been  several  studies  which  conclude 
that  procedural l zed  job  aids  improve  performance,  especially 
of  novices.  Both  Neiderman  (1988)  and  Swezey  (IQS'7)  refer 
to  a  1971  study  by  Elliott  and  Joyce,  in  which  high  school 
students,  provided  with  only  12  hours  of  training  and  a 
highly  pr ocedur a  1 i zed  troubleshooting  aid.  equalled  or 
exceeded  the  performance  of  experienced  Air  Force 
technicians  using  'conventional'  troubleshooting  methods. 
The  conclusion  drawn  from  this  and  similar  studies  is  that 
extensive  troubleshooting  training  is  not  necessary,  and 
considerable  training  time  and  money  can  be  saved  by  relying 
on  highly  pr ocedur a  1 1 zed  diagnostic  aids  instead  of 
cognitive  knowledge  to  solve  troubleshooting  problems  in  the 
field. 

However,  extensive  procedural l zat i on  has  been  found  to  be 
inadequate  in  implementation.  In  an  extensive  review  of 
Navy  fleet  maintenance  problems,  Nauta  (1985)  concluded  that 
procedural i zed  training  and  troubleshooting  aids  (i.e.  , 
manuals  and  test  equipment)  left  technicians  unprepared  to 
diagnose  many  of  the  faults  which  were  encountered, 
resulting  in  a  variety  of  ills:  job  dissatisfaction  and 
stagnation,  as  technicians  were  helpless  to  solve  many 
diagnostic  problems  and  did  not  have  the  basic  knowledge 
which  would  permit  learning  from  experience;  excessive 
consumption  of  components,  since  the  only  diagnostic 
strategy  available  to  the  technician  was  to  substitute  parts 
until  the  problem  disappeared;  and  excessive  reliance  on 
contractors  and  special  expert  teams  to  diagnose  equipment. 

While  reported  research  and  actual  experience  seem  to  be 
contradictory,  an  explanation  is  available.  Procedural i zed 
aids  d_o  permit  novice  technicians  to  perform  proficiently, 
when  they  are  available  and  accurate.  Experience  in  all  the 
services  is  that  traditional  troubleshooting  manuals  have 
been  inadequate:  they  are  often  delivered  late  and  are  full 
of  errors,  omissions,  and  inconsistencies.  Though  the 
quality  of  manuals  improves  over  time  as  errors  are 
corrected,  updates  are  slow  getting  to  the  field,  and  thus 
manuals  remain  out  of  step  with  engineering  changes  and 
equipment  modifications  throughout  the  life  of  the  system. 
More  troublesome  is  the  tendency  of  field  personnel  to  lose 
confidence  in  inaccurate  manuals,  and  therefore,  not  use 
them;  this  may  be  especially  true  of  novice  troubleshooters, 
who  do  not  have  the  training  or  experience  base  which  allows 
them  to  'fill  in  the  gaps'  of  misleading  technical 
information  (especially  if  they  received  only  procedural 
training) . 


2 


Although  improved  formats  for  procedural ued  troubleshooting 
aids  have  been  investigated  (Swezey  11987)),  developing  and 
validating  procedures  for  complex  equipment  will  remain 
difficult  ( Mo  rris  (198  5)  and  Swezey  (1987;).  Procedural ized 
aids  are  also  bulky  and  cumbersome,  which  can  inhibit  their 
effective  use  in  an  operational  environment.  More 
seriously,  such  reliance  upon  procedural  1  zed  aids  leads  to 
an  inability  to  perform  in  the  event  that  the  aid  is  not 
available,  is  m  error,  or  does  not  cover  a  situation 
encountered  on  the  actual  equipment.  This  is  precisely  the 
situation  that  the  Navy  found  to  exist  (Nauta  (1985)):  Early 
Built  In  Test  Equipment  (BITE) ,  designed  to  isolate  faults 
among  electronic  components,  failed  to  live  up  to 
expectations  and  design  goals  for  the  required  percentage  of 
faults  to  be  correctly  isolated.  Technical  manuals  lacked 
the  information  the  technicians  needed  to  isolate  the  faults 
presumably  handled  by  BITE.  Navy  procedural ized  training 
(by  design)  did  not  include  troubleshooting  training  which 
mignt  have  prepared  the  technicians  to  handle  unexpected 
situations:  only  expected  tasks  were  trained. 

However,  it  is  critical  that  weapons  systems  be  diagnosed 
and  repaired  quickly  for  novel  as  well  as  'routine' 
failures:  it  should  be  obvious  that  only  failures  which  were 
expected  by  the  design  and  maintenance  engineers  could  be 
covered  by  procedures.  The  diagnostician  must  have  some 
training  to  fall  back  on  in  such  situations. 

The  solution  called  for  by  the  Navy  was  more  cognitive  and 
theoretical  training.  They  also  undertook  the  EPICS 
(Enlisted  Personnel  Individualized  Career  System)  project, 
to  completely  integrate  training  and  personnel  management  of 
technicians,  and  to  develop  'enrirhed'  troubleshooting  job 
aids  which  allow  the  technician  to  develop  system  knowledge 
(Smillie  and  Clelland  (1986),  Smiiiie  (1985,  1986)).  Swezey 
i 1987)  also  concludes  that  training  is  necessary  to  prevent 
over-reliance  on  job  aids.  The  degree  to  which  the  training 
should  be  cognitive  or  procedural ized  has  also  been 
investigated. 

Morris  (1985)  reports  several  studies  which  conclude  that 
theoretical  information  does  no  t  lead  to  improved 
troubleshooting  performance,  even  though  technicians  often 
indicate  a  desire  for  more  theoretical  and  systems 
knowledge.  Novice  technicians  usually  cannot  develop 
effective  troubleshooting  strategies  from  theoretical 
information  without  guidance.  Research  indicates  that  it  is 
necessary  to  teach  students  how  to  us e  knowledge.  Techniques 
reported  by  Moms  (1985)  include  helping  the  student  to 
relate  a  schematic  to  the  actual  equipment,  instructing  the 
student  to  develop  a  troubleshooting  plan,  and  providing 
instruction  on  analyzing  symptoms.  He  also  reports  two 


3 


simulation  systems  which  allow  practice  of  troubleshooting 
skills:  the  Generalized  Maintenance  Trainer  Simulator  / 
Electronics  Equipment  Maintenance  Trainer  (GMAT/EEMT):  and 
SOPHIE  (the  SOPHisticated  Instructional  Environment) ,  an 
intelligent  interactive  computer  based  instructional  system 
which  allows  students  to  learn  systems  knowledge  and 
practice  troubleshooting  problems. 

There  is  also  evidence  that  more  experienced  troubleshooters 
(Morris  (1985))  are  able  to  learn  more  readily  on  the  job 
than  are  novices,  and  thus  gain  more  transfer  of  knowledge 
from  performance  aids  because  they  are  able  to  perceive  the 
underlying  troubleshooting  principles. 

The  reasonable  conclusion  is  that  a  combination  of  training 
methods  is  best.  The  proportions  will  vary  by  technician 
specialty,  the  skill  level  being  trained,  and  task 
requirements  of  specific  weapons  systems.  Some  theoretical 
information  (principles  of  electronics)  ,  systems  knowledge 
(how  the  system  operates)  ,  and  troubleshooting  training  (how 
to  apply  such  knowledge  to  develop  tr oub leshooting  and 
testing  strategies)  should  be  provided  to  diagnosticians,  as 
well  as  hands-on  step-by-step  training  and  practice.  Even 
with  improved  on-board  diagnostics  (BITE) ,  and  even  if 
procedural lzed  aids  are  available  to  assist  the  proficiency 
of  novice  diagnosticians  and  serve  as  memory  aids  for 
intermediate  and  experienced  diagnosticians,  it  is  unlikely 
that  100%  diagnosis  could  be  attained  without  reliance  upon 
human  problem  solving  in  uncertain  situations  and  for  novel 
failures  when  it  is  necessary  to  reason  from  first 
principles.  The  diagnostician  must  rely  upon  transf errable 
knowledge  and  skills  to  solve  problems  which  could  not  be 
solved  ahead  of  time  by  the  design  of  procedural lzed  aids. 

Reas  o  n 1 ng : 

Cognitive  theory  (Oden  (1987))  postulates  that  human 
learning  and  reasoning  are  based  upon  man's  ability  to 
construct  mental  models,  which  can  be  simulated  to  determine 
what  would  happen  under  various  hypotheses.  These  models 
need  not  be  complete  or  sophisticated  to  be  powerful.  An 
implication  for  the  design  of  man-computer  interfaces  is 
that  the  system  might  have  to  adapt  to  the  human’s  mental 
model,  Oden  concludes  that  work  in  a  variety  of  disciplines 
(i.e.,  psychology  and  artificial  intelligence)  supports 
acceptance  of  models. 

Not  surprisingly,  research  in  the  domain  of  diagnostics 
supports  the  concept  of  mental  models.  Successful 
diagnosticians  use  a  mental  model  of  the  system  (based  upon 
structure  and  function)  to  determine  the  logical  site  of  a 
fault.  According  to  research  cited  by  Lee  (1988)  .  the 
expert  is  more  proficient  than  the  novice  because  the  expert 
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has  the  ability  to  construct  a  more  correct  mental  model  of 
the  system,  and  knows  which  model  to  use  for  different 
situations.  The  novice  tends  to  formulate  a  naive  mental 
model  which  may  miss  key  functional  interrelationships  (Gott 
1988) .  This  explains  the  tendency  of  the  novice  to 
troubleshoot  in  physical  proximity  of  the  symptom,  rather 
than  the  expert’s  (correct)  approach  to  troubleshoot 
components  in  close  functional  relationship  to  the  symptom, 
regardless  of  the  physical  location  of  those  components. 
Lee  contrasts  model  representations  drawn  by  experts  and 
novices  to  explain  their  troubleshooting  strategies. 

Evidence  suggests  that  experts  as  well  as  novices  may  employ 
associative  (symptom  to  fault)  troubleshooting  "rules," 
based  upon  experience  or  heuristics.  But  if  these  fail,  the 
expert  falls  back  upon  his  mental  model  and  uses  causal  or 
"deep"  reasoning  to  diagnose  the  fault  (Yoon  (1988)).  This 
solution  technique  must  be  employed  for  novel  failures.  The 
novice,  in  contrast,  has  an  inadequate  understanding  of  the 
causal  relationships  in  the  system  and  employs  an 
inefficient  diagnosis  strategy  (i.e.  working  backward  from 
the  location  of  the  symptom).  quickly  overburdening  his 
short-term  memory  (especially  in  a  stressful  situation). 
This  leads  to  confusion  and  frustration. 

Research  thus  explains  the  inability  of  early  diagnostic 
expert  systems  to  function  as  robustly  as  expected.  Expert 
systems  based  upon  rules  alone  are  often  brittle  and  fail 
rapidly  for  unexpected  faults  or  unanticipated  combinations 
of  symptoms  (Lee  (1988)).  Such  systems  mimic  only  the 
portion  of  the  expert's  approach  which  is  based  upon 
"shallow"  knowledge.  For  novel  situations,  the  expert 
relies  upon  causal  knowledge  which  is  not  available  in  a 
rule-based  expert  system.  Experts  also  employ  their  mental 
models  in  an  automatic  mode  after  extensive  experience,  and 
may  not  even  be  able  to  articulate  the  methods  they  are 
actually  employing  (Gott  (1988)).  Traditional  expert 
systems  also  cannot  readily  explain  conclusions  to  the 
expert  or  novice  (for  validation  or  learning,  respectively). 

A  powerful  model  methodology  is  described  by  Yoon  (1988) . 
The  interface  between  the  human  and  the  computer  is  designed 
to  solve  the  problem  identified  by  Swezey  (1987)  for  all  job 
aids:  the  psychological  needs  of  people.  "The  motivational 
properties  of  leading  individuals  by  the  hand  through  a 
series  of  steps  without  requiring  them  to  exercise  their  own 
judgement  and/or  logic  is  typically  considered  demeaning... 
Job  aids  must  be  designed  in  a  fashion  that  alleviates  the 
feelings  (of)  job  incumbents  that  they  are  merely  ’trained 
monkeys'  who  are  qualified  only  to  perform  the  simplest 
tasks."  This  approach  is  especially  insulting  when  the 
human’s  tasks  are  directed  by  the  computer'  Yoon  (1988) 
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points  out  that  traditional  expert  systems  often  use  a 
computer -d 1 rec ted  serial  interaction.  Mot  only  does  this 
prevent  the  human  from  employing  any  strategy  of  his  own, 
but  the  serial  nature  of  the  interaction  interrupts  his 
information  processing.  preventing  deep  reasoning.  As 
discussed  above,  traditional  diagnostic  expert  systems 
typically  fall  short  in  novel  situations.  A  directed 
interface  reinforces  that  failure  by  discouraging  human 
problem  solving.  Yoon  suggests  an  independent  or  suggestive 
form  of  interaction  based  upon  the  philosophy  that  the 
system  should  provide  support  for  the  human’s  information 
processing  needs:  to  prevent  cognitive  and  short-term  memory 
overload,  and  to  overcome  we  1 1 -documented  tendencies  of 
human  problem  solvers  to  under - s pec i f y  the  hypotheses  set 
(based  upon  cognitive  limitations)  and  to  fail  to 
incorporate  negative  evidence.  This  is  a  different 
philosophy  than  that  of  many  current  systems,  which  presume 
the  system  to  perform  the  diagnosis  in  conjunction  with 
symptom  information  and  test  observations  provided  by  the 
human.  Mot  only  is  this  potentially  unsatisfying,  the 
operator  will  not  develop  diagnostic  skills  that  may  become 
necessary  when  the  automation  fails  (Yoon  (1988)).  This 
realization  is  fully  consistent  with  the  conclusions 
regarding  diagnostic  knowledge  and  training  requirements. 

Thus,  the  design  of  diagnostic  job  performance  aids  requires 
creativity  and  attention  to  the  human  interface.  Some 
prevision  for  automatic  diagnosis  will  be  necessary:  for 
example,  to  reach  a  conclusion  on-board  without  the  need  for 
skilled  technicians;  to  ensure  the  proficiency  of  novice 
technicians;  or  to  save  diagnosis  time,  especially  in 
stressful  situations  of  fatigue  or  time  pressure,  when  the 
priority  for  getting  the  system  operational  as  fast  as 
possible  mitigates  the  technician's  desire  to  control  or 
learn  from  that  particular  diagnostic  session.  However,  the 
diagnostic  aid  must  be  designed  for  learning  and  explanation 
as  much  as  for  internal  diagnostic  reasoning. 

The  recent  trend  in  the  development  of  diagnostic  systems  is 
to  employ  causal  models  of  the  system  at  one  or  more  levels 
of  abstraction.  Not  only  is  such  an  approach  appealing  in 
that  it  might  "model"  the  way  a  human  diagnostician  sees  the 
device  (hence,  a  consistent  model  sharing  between  the 
diagnostic  software  and  the  human) ,  it  also  allows  natural 
explanation  and  training.  BBN  (Feurzig  (1988))  has 
developed  a  system  whioh  can  teach  electrical  systems 
troubleshooting  theory  and  perform  diagnosis  using  a  model 
of  the  system  with  qualitative  reasoning.  The  system  has  a 
model  of  the  student  (level  of  experience,  etc.)  and  a  model 
of  the  domain.  It  also  helps  the  student  to  develop  his  own 
model  of  the  domain  and  knowledge  which  is  trans f err ab 1 e 
from  one  diagnostic  situation  to  another.  As  suggested  by 
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Yoon  (1988)  ,  one  method  of  interaction  between  human  and 
system  is  for  each  to  follow  an  independent  diagnostic 
strategy.  A  causal  model  can  be  simulated  qualitatively  to 
predict  system  functioning  or  to  compare  normal  and  observed 
system  behavior  to  isolate  a  faulty  portion  of  the 
structure.  Yoon's  research  shows  that  such  information  is 
compatible  with  the  human's  reasoning  process.  Model -based 
systems  are  superior  for  automatic  diagnosis  because  they 
are  not  limited  to  symptoms  or  faults  which  were  anticipated 
(as  are  rule  based  systems),  but  can  reason  from  the  model 
in  novel  situations.  Another  advantage  of  models  is  the 
potential  to  use  the  CAD  model  of  the  system  generated  at 
design  time,  saving  expensive  software  development. 

OTHER  DIAGNOSTIC  AID  DESIGN  CONSIDERATIONS 

The  conceptual  framework  for  diagnosis  aiding,  based  upon 
the  above  research,  provides  the  basis  for  evaluating 
function  allocation  between  man  and  machine  (aid) ,  and  the 
man-machine  interface.  To  assess  the  ability  of  the  aid  to 
function  in  the  operational  environment,  attention  to  the 
design  of  hardware  is  necessary.  Novel  (as  well  as  existing 
commercial)  implementations  should  be  explored,  based  upon 
current  developments  in  the  fields  of  natural  language 
understanding  and  voice  recognition  (Simpson  (1985), 
Ha up t man  (1988),  Jones  (1989),  Leiser  (1989),  Martin  (1989), 
Swezey  ( 1987) ) ,  human -computer  interfaces  (Farooq  ( 1989) , 
Morrison  (1988),  Shneiderman  (1988)),  display  devices  such 
as  virtual  images  (Newmar.  (1980,  1984,  1987,1987a)),  and 
paper  versus  electronic  delivery  (Smiliie  (1988)  ,  Nugent 
(1987);,  Swezey  (1987)  reports  on  projects  undertaken  in 
voice  and  virtual  image  interfaces  which  have  potential  to 
free  the  technician's  hands. 

Expectations  for  the  portable  rr.a intenance  aid  are  likely  to 
meet  with  disappointment,  unless  concurrent  improvements  in 
other  areas  which  affect  system  diagnostic  performance  are 
also  made  (Nauta  (1985)).  The  biggest  factor  affecting 
diagnostic  performance  is  the  design  of  the  weapons  system 
itself.  Although  this  is  not  a  variable  which  can  be 
altered  for  existing  weapons  systems,  it  is  vital  that  such 
relationships  be  understood,  and  that  computer-based 
maintenance  aids  not  be  perceived  as  a  solution  for  more 
systemic  problems. 

CONCLUSIONS 

Research  Issues : 

Further  research  is  required;  however,  the  perceived  need 
for  immediate  assistance  and  the  advertised  availability  of 
solutions  dictate  that  diagnostic  aids  will  be  developed  and 
implemented  before  the  domain  is  fully  understood.  'rms  is 
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to  be  expected,  but,  the  warning  of  Morris  (1985)  must  be 
heeded:  ‘accompanying  this  trend  toward  greater  use  of 
technology,  there  seems  to  have  been  a  shift  in  emphasis 
that  avoided,  wh  ether  purposefully  or  incidentally,  an 
evaluation  of  transfer  of  training.  All  too  often  recent 
’experiments’  have  been  demonstrations  rather  than 
evaluations.  Research  should  retain  as  its  primary  goal  the 
evaluation  of  transfer  to  real-life  situations.' 

In  addition,  earlier  research  on  troubleshooting  is  often 
based  upon  a  number  of  students  identifying  faults  in  a 
diagrammed  network,  leading  to  conclusions  about 
proceduralization  and  level  of  cognitive  skills  necessary  to 
perform  troubleshooting.  While  many  of  these  experiments 
used  valid  scientific  techniques,  they  do  not  usually  take 
into  account  the  real  world,  where  errors  in  procedural  1  zed 
aids  are  possible,  and  where  systems  are  so  complex  that 
such  problems  might  be  considered  as  "toy'  -  i.e.  results 
might  not  be  transferrable. 

While  it  is  possible  to  make  informed  conjectures  from 
existing  research.  it  is  not  yet  kno  wn  wh  at  level  of 
cognitive  skill  is  necessary  for  diagnosis  and  how  it  should 
be  provided.  (Readers  are  recommended  to  the  Neiderman 
(1988)  and  Morris  (1985)  surveys  to  judge  for  themselves  the 
implications  of  the  limited  and  sometimes  contradictory 
research  results  that  are  presently  available) . 

Design  Problem : 

The  problem  in  diagnostic  job  aiding,  then,  is  to  develop  a 
diagnostic  framework  consistent  with  human  reasoning  methods 
and  cognitive  limitations,  and  to  design  diagnostic 
information  aiding  within  that  framework  by  parceling  job 
knowledge  requirements  among  training  (human  long  term 
memory) ,  human  short  term  memory,  and  the  diagnostic  aid. 
The  literature  supports  the  conclusion  that  model-based 
approaches  are  the  most  natural  and  potentially  the  most 
supportive  of  both  the  human  diagnostic  process  and 
diagnostician  learning,  as  well  as  offering  the  most 
powerful  appi-oach  for  machine- independent  diagnosis  (when 
necessary  or  desirable).  Designing  the  diagnostic  job 
performance  aid  begins  with  these  decisions.  Hardware  and 
software  to  house  and  deliver  information  aiding  must  then 
be  designed  to  meet  the  operator's  needs  m  the  operational 
environment.  Design  principles  from  a  variety  of  technical 
disciplines  must  be  considered. 
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